算式识别(挑战项目)

描述

使用深度学习识别一张图片中的算式。

  • 输入:一张彩色图片
  • 输出:算式的内容

数据

数据集下载链接:

https://s3.cn-north-1.amazonaws.com.cn/static-documents/nd009/MLND+Capstone/Mathematical_Expression_Recognition_train.zip

此数据集包含10万张图片,每张图里面都有一个算式。

  • 可能包含 +-* 三种运算符
  • 可能包含一对括号,也可能不包含括号
  • 每个字符都可能旋转,所以 + 号可能长得像我们平时手写的 * 号,不过 * 号有六个瓣
In [1]:
import pandas as pd
import matplotlib.pyplot as plt
import cv2
import sys
import numpy as np
import os
import datetime
from keras import backend as K
from keras.layers.convolutional import Conv2D, MaxPooling2D
from keras.layers import Input, Dense, Activation, BatchNormalization, Dropout
from keras.layers import Reshape, Lambda
from keras.layers.merge import add, concatenate
from keras.models import Model
from keras.optimizers import Adam
from keras.layers.recurrent import GRU
from keras.preprocessing import image
import keras.callbacks
from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle
%matplotlib inline
Using TensorFlow backend.

查看数据集的情况,包含图片路径和内容标签

In [2]:
csv_path = 'train.csv'
df = pd.read_csv(csv_path)
df.head(5)
Out[2]:
filename label
0 train/0.jpg (0+0)+9=9
1 train/1.jpg 9*8+6=78
2 train/2.jpg 1+7+0=8
3 train/3.jpg 5+4+9=18
4 train/4.jpg 7-(1*7)=0

显示图片和旋转图片的效果

In [3]:
def show_imgs(imgs, labels):       
    for i in range(len(imgs)): 
        plt.title(labels[i])
        img = imgs[i].transpose(1, 0, 2)
        plt.imshow(img)
        plt.show()
        h, w, _ = img.shape
        img = image.random_rotation(img,
                                    3 * (w - h//5) / w + 1,
                                    row_axis=0,
                                    col_axis=1,
                                    channel_axis=2,)
        plt.imshow(img)
        plt.show()
        
def load_imgs(series):
    total = len(series)
    imgs = np.empty((len(series), width, height, depth),dtype=np.uint8)
    for i,img_path in enumerate(series):     
        if ((i+1) % 100) == 0:
            print("\rImg Processing: {:,}/{:,}".format(i+1, total), end="")
            sys.stdout.flush()
        bgr_img = cv2.imread(img_path)
        rgb_img = cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB)
        imgs[i] = rgb_img.transpose(1, 0, 2)
    return imgs

nums = 5
img = cv2.imread(df['filename'][0])
height, width, depth = img.shape
imgs = load_imgs(df['filename'][:nums])
labels = df['label'][:nums]
print(imgs.shape)
show_imgs(imgs[:nums],labels)   

del df
(5, 300, 64, 3)

定义SaveCallback类,训练完成后的回调函数,方便保留weight和loss记录

In [4]:
class SaveCallback(keras.callbacks.Callback):
    def __init__(self, train_time):
        self.output_dir = train_time
        if os.path.exists(self.output_dir) == False:
            os.makedirs(self.output_dir)
            
    def on_epoch_end(self, epoch, logs=None):
        self.model.save_weights(os.path.join(self.output_dir, "weights_{:03d}.h5".format(epoch + 1)))
        log_path = os.path.join(self.output_dir, 'log.csv')        
        logs['epoch'] = epoch        
        if epoch == 0:
            df = pd.DataFrame(logs, index=[0])
        else:
            df = pd.read_csv(log_path)
            df = pd.concat([df,pd.DataFrame(logs, index=[0])])     
        if df.iloc[-1]['loss'] <= df['loss'].min():
            self.model.save_weights(os.path.join(self.output_dir, "model_best_weights.h5"))
        df.to_csv(log_path, index=False)                                             

定义ImageSet类,划分训练,验证,测试集,以及处理产生对应的数据

In [5]:
def get_ImgData(img_path, rotate=False):
    img = cv2.imread(img_path).transpose(1, 0, 2)
    w, h, _ = img.shape    
    if rotate:
        img = image.random_rotation(img, 
                                    3 * (w - h // 5) / w + 1,
                                    row_axis=0,
                                    col_axis=1,
                                    channel_axis=2,)
    img = img.astype(np.float32) / 255
    return img

def sample_label_split(df):
    return df['filename'], df['label']
    
class ImageSet(keras.callbacks.Callback):
    def __init__(self, minibatch_size, start_epoch):
        self.minibatch_size = minibatch_size         
        
        DATASET_PATH = 'data_set'
        train_path = os.path.join(DATASET_PATH, 'train_set.csv')
        valid_path = os.path.join(DATASET_PATH, 'valid_set.csv')
        test_path = os.path.join(DATASET_PATH, 'test_set.csv')
        
        self.random_seed = 50
        np.random.seed(self.random_seed)

        df = pd.read_csv(csv_path)        
        if os.path.exists(DATASET_PATH) == False:             
            shuffle(df, replace=True, random_state=self.random_seed)
            df_train, df_test = train_test_split(df, test_size=0.2, random_state=self.random_seed)
            df_train, df_valid = train_test_split(df_train, test_size=0.25, random_state=self.random_seed)
            os.makedirs(DATASET_PATH)
            df_train.to_csv(train_path, index=False)
            df_valid.to_csv(valid_path, index=False)
            df_test.to_csv(test_path, index=False)
            print("Split data set done!")
            
        characters = set()
        y = sample_label_split(df)[1]
        for label in y:
            characters.update(label)
        self.characters = ''.join(sorted(characters))
        self.n_class = len(self.characters) + 1
        self.max_string_len = max(map(len, y))
        
        self.df_train = pd.read_csv(train_path)
        self.df_valid = pd.read_csv(valid_path)
        self.df_test = pd.read_csv(test_path)
        self.X_train, self.y_train = sample_label_split(self.df_train)
        self.X_valid, self.y_valid = sample_label_split(self.df_valid)
        self.X_test, self.y_test = sample_label_split(self.df_test)
        self.image_shape = cv2.imread(self.X_train[0]).transpose(1, 0, 2).shape        
        
        self.cur_train_index = 0
        self.cur_valid_index = 0
        self.cur_test_index = 0
        self.max_train_index = self.df_train.shape[0]
        self.max_valid_index = self.df_valid.shape[0]
        self.max_test_index = self.df_test.shape[0]
        
        self.getImgData = lambda img_path: get_ImgData(img_path,rotate=False)
        
        def plot_char_freq(serise, title):
            char_dict = {}
            const_freq = 1
            for label in serise:
                for ch in label:
                    char_dict[ch] = char_dict.get(ch, 0) + const_freq   
            for k in char_dict.keys():
                char_dict[k] /=  float(serise.shape[0])
            k_list, v_list = [], []
            for k,v in sorted(char_dict.items(), key=lambda kv: kv[1], reverse=True):
                k_list.append(k)
                v_list.append(v)
            plt.bar(k_list, v_list)
            plt.title(title)
            plt.show()
            
        if start_epoch == 0:
            print('train nums:', self.df_train.shape[0])
            print('valid nums:', self.df_valid.shape[0])
            print('test nums:', self.df_test.shape[0])
            plot_char_freq(self.y_train, 'train_set')
            plot_char_freq(self.y_valid, 'valid_set')
            plot_char_freq(self.y_test, 'test_set')
    
    def set_input_length(self, length):
        self.input_length = length
    
    def get_minibatch(self, index, size, X, y):
        X_data = np.ones([size, *self.image_shape])
        labels = np.ones([size, self.max_string_len]) * -1
        input_length = np.zeros([size, 1])
        label_length = np.zeros([size, 1])
        text_label = np.zeros([size], dtype=np.str_)
                
        for i in range(size):
            X_data[i] = self.getImgData(X[i + index])
            labels[i, :len(y[i + index])] = [self.characters.find(c) for c in y[i + index]]
            input_length[i] = self.input_length - 2
            label_length[i] = len(y[i + index])
            text_label[i] = y[i + index]
        # respond to Input name    
        inputs = { 'the_input': X_data,
                 'the_labels': labels,
                 'input_length': input_length,
                 'label_length': label_length,
                 'text_label': text_label            
        }        
        outputs = {'ctc': np.zeros(size)}    
        return inputs, outputs
    
    def next_train(self):
        while True:
            ret = self.get_minibatch(self.cur_train_index,
                                 self.minibatch_size, self.X_train, self.y_train )
            self.cur_train_index += self.minibatch_size
            if (self.cur_train_index + self.minibatch_size) > self.max_train_index:                
                shuffle(self.df_train, replace=True, random_state=self.random_seed)
                self.X_train, self.y_train = sample_label_split(self.df_train)
                self.cur_train_index =  0
            yield ret
    
    def next_valid(self):
        while True:
            ret = self.get_minibatch(self.cur_valid_index,
                                 self.minibatch_size, self.X_valid, self.y_valid )
            self.cur_valid_index += self.minibatch_size
            if (self.cur_valid_index + self.minibatch_size) > self.max_valid_index:
                shuffle(self.df_valid, replace=True, random_state=self.random_seed)
                self.X_valid, self.y_valid = sample_label_split(self.df_valid)
                self.cur_valid_index =  0
            yield ret
            
    def next_test(self, size):
        while True:
            ret = self.get_minibatch(self.cur_test_index,
                                 size, self.X_test, self.y_test)
            self.cur_test_index += size
            if (self.cur_test_index + size) > self.max_test_index:
                self.cur_test_index = 0
                break
            yield ret       
            
    def on_train_begin(self, logs=None):
        self.getImgData = lambda img_path: get_ImgData(img_path,rotate=False)

    def on_epoch_begin(self, epoch, logs=None):
        if epoch >= 20:
            self.getImgData = lambda img_path: get_ImgData(img_path,rotate=True)
In [6]:
def ctc_lambda_func(args):
    y_pred, labels, input_length, label_length = args
    # the 2 is critical here since the first couple outputs of the RNN
    # tend to be garbage:
    y_pred = y_pred[:, 2:, :]
    return K.ctc_batch_cost(labels, y_pred, input_length, label_length)
def ctc_loss(y_true, y_pred):
    return y_pred
In [7]:
def draw_model_detail(model):
    from keras.utils import plot_model
    plot_model(model, to_file='model.png')
    from IPython.display import SVG
    from keras.utils.vis_utils import model_to_dot
    SVG(model_to_dot(model).create(prog='dot', format='svg'))    
    model.summary()

定义训练模型的函数,包含模型,模型训练,训练历史数据展示和正确率验证

In [21]:
def train(train_time, start_epoch, stop_epoch, lr, test_proc=False, use_epoch=0, simple_mode=False, drop_out=0.4, test_show=False):
    imgs_per_epoch = 80000
    valid_split = 0.25
    valid_images = int(imgs_per_epoch * valid_split)
    train_images = imgs_per_epoch - valid_images

    pool_size = (2, 2)
    rnn_size = 256
    act = 'relu'
    k_init = 'he_normal'
    time_dense_size = 128
    minibatch_size = 32
    padding = 'same'

    img_set = ImageSet(minibatch_size, start_epoch)
    input_data = Input(name='the_input', shape=img_set.image_shape, dtype='float32')    
    if simple_mode:
        inner = Conv2D(16, (2, 2), padding=padding)(input_data)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = MaxPooling2D(pool_size=pool_size)(inner)
        
        inner = Conv2D(32, (2, 2), padding=padding)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = MaxPooling2D(pool_size=pool_size)(inner)
        
        inner = Conv2D(64, (2, 2), padding=padding)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = MaxPooling2D(pool_size=pool_size)(inner)

    else:
        inner = Conv2D(32, (5, 5), padding=padding, kernel_initializer=k_init)(input_data)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = Conv2D(32, (5, 5), padding=padding, kernel_initializer=k_init)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = MaxPooling2D(pool_size=pool_size)(inner)

        inner = Conv2D(64, (4, 4), padding=padding, kernel_initializer=k_init)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = Conv2D(64, (4, 4), padding=padding, kernel_initializer=k_init)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = MaxPooling2D(pool_size=pool_size)(inner)

        inner = Conv2D(128, (3, 3), padding=padding, kernel_initializer=k_init)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = Conv2D(128, (3, 3), padding=padding, kernel_initializer=k_init)(inner)
        inner = BatchNormalization()(inner)
        inner = Activation(act)(inner)
        inner = MaxPooling2D(pool_size=pool_size)(inner)

    inner_shape = inner.get_shape().as_list()
    inner = Reshape(target_shape=(inner_shape[1], inner_shape[2] * inner_shape[3]))(inner)
    img_set.set_input_length(inner_shape[1])

    inner = Dense(time_dense_size, kernel_initializer=k_init)(inner)
    inner = BatchNormalization()(inner)
    inner = Activation(act)(inner)

    gru_1 = GRU(rnn_size, return_sequences=True, kernel_initializer=k_init, name='gru1')(inner)
    gru_1b = GRU(rnn_size, return_sequences=True, go_backwards=True, kernel_initializer=k_init, name='gru1_b')(
        inner)
    gru1_merged = add([gru_1, gru_1b])

    gru_2 = GRU(rnn_size, return_sequences=True, kernel_initializer=k_init, name='gru2')(gru1_merged)
    gru_2b = GRU(rnn_size, return_sequences=True, go_backwards=True, kernel_initializer=k_init, name='gru2_b')(
        gru1_merged)
    inner = Dropout(drop_out)(concatenate([gru_2, gru_2b]))
    y_pred = Dense(img_set.n_class, kernel_initializer=k_init, activation='softmax')(inner)
    pred_model = Model(input=input_data, output=y_pred)
    
    labels = Input(name='the_labels', shape=[img_set.max_string_len], dtype='float32')
    input_length = Input(name='input_length', shape=[1], dtype='int64')
    label_length = Input(name='label_length', shape=[1], dtype='int64')

    loss_out = Lambda(ctc_lambda_func, name='ctc', output_shape=[1])([y_pred, labels, input_length, label_length])

    model = Model(inputs=(input_data, labels, input_length, label_length), outputs=loss_out)
    model.compile(loss={'ctc': ctc_loss}, optimizer=Adam(lr))
    
    if start_epoch < stop_epoch:
        save_cb = SaveCallback(train_time)
        if start_epoch > 0:
            model.load_weights(os.path.join(train_time, "weights_{:03d}.h5".format(start_epoch - 1)))
        else:
            draw_model_detail(model)
        model.fit_generator(generator=img_set.next_train(),
                                      steps_per_epoch=train_images // img_set.minibatch_size,
                                      epochs=stop_epoch,

                                      validation_data=img_set.next_valid(),
                                      validation_steps=valid_images // img_set.minibatch_size,

                                      callbacks=[
                                          keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.1, patience=5, verbose=2, mode='min'),
                                          img_set,
                                          save_cb,
                                      ],
                                      initial_epoch=start_epoch,
                                      verbose=2)
    log_path = os.path.join(train_time, 'log.csv')   
    history = pd.read_csv(log_path)
    # 绘制训练 & 验证的损失值
    plt.plot(history['loss'])
    plt.plot(history['val_loss'])
    plt.title('Model loss')
    plt.ylabel('Loss')
    plt.xlabel('Epoch')
    plt.legend(['Train', 'valid'], loc='upper right')
    plt.show()


    plt.plot(history['lr'])
    plt.title('Model lr')
    plt.ylabel('learning rate')
    plt.xlabel('Epoch')
    plt.legend(['lr'], loc='upper right')
    plt.show()
    
    if test_proc == False:
        return
    
    if use_epoch :
        model.load_weights(os.path.join(train_time, "weights_{:03d}.h5".format(use_epoch)))
    else:
        model.load_weights(os.path.join(train_time, 'model_best_weights.h5'))  
    
    def label_to_text(labels):
        return ''.join([img_set.characters[int(x)] for x in labels if x > -1])
        
    acc = []
    cnt = 0
    batch_size = 100
    for inputs, _ in img_set.next_test(batch_size):
        X_test, y_test = inputs['the_input'], inputs['the_labels']
        y_pred = pred_model.predict(X_test)
        shape = y_pred[:,2:,:].shape      
        out = K.get_value(K.ctc_decode(y_pred[:,2:,:], input_length=np.ones(shape[0])*shape[1])[0][0])[:, :img_set.max_string_len]      
        cur_sum = 0
        for img, y, pre in zip(X_test, y_test, out):
            l = len([x for x in pre if x > -1])
            if (y[:l] == pre[:l]).all() and test_show==False:
                cur_sum += 1
            else:
                img = (img * 255).astype(np.uint8).transpose(1, 0, 2)
                plt.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
                plt.title('y_test = {} : y_pred ={}'.format(label_to_text(y),label_to_text(pre)))
                plt.show()
        if test_show: break
        acc.append(cur_sum/(len(X_test)))
        cnt += len(X_test)
    if test_show == False:
        print(acc)
        print('test_acc: {:%}'.format(np.mean(acc)))

可以观察训练集、验证集和测试集字符的分布情况。结果是分布基本一致。

  • "=" 100%

  • "-" 其次

  • "+*()" 差不多

  • "1" 数字里最多

  • 其余依序数字差别不大 (仅验证集的5,6顺序与其他不同,但是本身出现的机率相差并不悬殊)

In [9]:
train_time = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
In [10]:
train(train_time, 0, 10, 1e-04)
train nums: 60000
valid nums: 20000
test nums: 20000
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1238: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1204: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1340: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
the_input (InputLayer)          (None, 300, 64, 3)   0                                            
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 300, 64, 32)  2432        the_input[0][0]                  
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 300, 64, 32)  128         conv2d_1[0][0]                   
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 300, 64, 32)  0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 300, 64, 32)  25632       activation_1[0][0]               
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 300, 64, 32)  128         conv2d_2[0][0]                   
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 300, 64, 32)  0           batch_normalization_2[0][0]      
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)  (None, 150, 32, 32)  0           activation_2[0][0]               
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 150, 32, 64)  32832       max_pooling2d_1[0][0]            
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 150, 32, 64)  256         conv2d_3[0][0]                   
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 150, 32, 64)  0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 150, 32, 64)  65600       activation_3[0][0]               
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 150, 32, 64)  256         conv2d_4[0][0]                   
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 150, 32, 64)  0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)  (None, 75, 16, 64)   0           activation_4[0][0]               
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 75, 16, 128)  73856       max_pooling2d_2[0][0]            
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 75, 16, 128)  512         conv2d_5[0][0]                   
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 75, 16, 128)  0           batch_normalization_5[0][0]      
__________________________________________________________________________________________________
conv2d_6 (Conv2D)               (None, 75, 16, 128)  147584      activation_5[0][0]               
__________________________________________________________________________________________________
batch_normalization_6 (BatchNor (None, 75, 16, 128)  512         conv2d_6[0][0]                   
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 75, 16, 128)  0           batch_normalization_6[0][0]      
__________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)  (None, 37, 8, 128)   0           activation_6[0][0]               
__________________________________________________________________________________________________
reshape_1 (Reshape)             (None, 37, 1024)     0           max_pooling2d_3[0][0]            
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 37, 128)      131200      reshape_1[0][0]                  
__________________________________________________________________________________________________
batch_normalization_7 (BatchNor (None, 37, 128)      512         dense_1[0][0]                    
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 37, 128)      0           batch_normalization_7[0][0]      
__________________________________________________________________________________________________
gru1 (GRU)                      (None, 37, 256)      295680      activation_7[0][0]               
__________________________________________________________________________________________________
gru1_b (GRU)                    (None, 37, 256)      295680      activation_7[0][0]               
__________________________________________________________________________________________________
add_1 (Add)                     (None, 37, 256)      0           gru1[0][0]                       
                                                                 gru1_b[0][0]                     
__________________________________________________________________________________________________
gru2 (GRU)                      (None, 37, 256)      393984      add_1[0][0]                      
__________________________________________________________________________________________________
gru2_b (GRU)                    (None, 37, 256)      393984      add_1[0][0]                      
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 37, 512)      0           gru2[0][0]                       
                                                                 gru2_b[0][0]                     
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 37, 512)      0           concatenate_1[0][0]              
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 37, 17)       8721        dropout_1[0][0]                  
__________________________________________________________________________________________________
the_labels (InputLayer)         (None, 11)           0                                            
__________________________________________________________________________________________________
input_length (InputLayer)       (None, 1)            0                                            
__________________________________________________________________________________________________
label_length (InputLayer)       (None, 1)            0                                            
__________________________________________________________________________________________________
ctc (Lambda)                    (None, 1)            0           dense_2[0][0]                    
                                                                 the_labels[0][0]                 
                                                                 input_length[0][0]               
                                                                 label_length[0][0]               
==================================================================================================
Total params: 1,869,489
Trainable params: 1,868,337
Non-trainable params: 1,152
__________________________________________________________________________________________________
Epoch 1/10
 - 1313s - loss: 6.5431 - val_loss: 1.4669
Epoch 2/10
 - 618s - loss: 0.5528 - val_loss: 0.3105
Epoch 3/10
 - 617s - loss: 0.2325 - val_loss: 0.2349
Epoch 4/10
 - 618s - loss: 0.1306 - val_loss: 0.1499
Epoch 5/10
 - 619s - loss: 0.0798 - val_loss: 0.1198
Epoch 6/10
 - 619s - loss: 0.0500 - val_loss: 0.0823
Epoch 7/10
 - 620s - loss: 0.0375 - val_loss: 0.0821
Epoch 8/10
 - 620s - loss: 0.0347 - val_loss: 0.0754
Epoch 9/10
 - 618s - loss: 0.0243 - val_loss: 0.0870
Epoch 10/10
 - 618s - loss: 0.0250 - val_loss: 0.1140

继续训练数据

In [22]:
train_time = '20190305-185535'
In [12]:
train(train_time, 10, 20, 2e-05, test_proc=True)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
Epoch 11/20
 - 622s - loss: 0.0068 - val_loss: 0.0359
Epoch 12/20
 - 616s - loss: 0.0037 - val_loss: 0.0337
Epoch 13/20
 - 617s - loss: 0.0029 - val_loss: 0.0346
Epoch 14/20
 - 616s - loss: 0.0021 - val_loss: 0.0409
Epoch 15/20
 - 616s - loss: 0.0023 - val_loss: 0.0377
Epoch 16/20
 - 616s - loss: 0.0019 - val_loss: 0.0361
Epoch 17/20
 - 617s - loss: 0.0014 - val_loss: 0.0367
Epoch 18/20
 - 616s - loss: 0.0011 - val_loss: 0.0358
Epoch 19/20
 - 616s - loss: 0.0023 - val_loss: 0.0366
Epoch 20/20
 - 616s - loss: 0.0012 - val_loss: 0.0346
[1.0, 0.99, 0.99, 1.0, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.96, 1.0, 0.98, 0.99, 1.0, 0.99, 0.99, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.96, 0.97, 0.98, 1.0, 1.0, 0.99, 0.99, 0.99, 0.98, 0.99, 1.0, 1.0, 0.98, 0.98, 0.99, 0.98, 1.0, 0.99, 0.99, 0.97, 0.98, 0.99, 0.99, 0.99, 1.0, 1.0, 0.99, 0.99, 1.0, 0.99, 0.99, 0.98, 0.99, 0.99, 0.99, 1.0, 0.98, 0.99, 0.98, 1.0, 0.98, 0.99, 1.0, 0.99, 1.0, 0.98, 0.98, 0.99, 0.99, 0.99, 0.97, 0.98, 0.99, 1.0, 1.0, 1.0, 0.98, 1.0, 0.97, 0.98, 1.0, 0.99, 0.99, 0.98, 0.98, 0.98, 1.0, 1.0, 1.0, 0.99, 0.98, 1.0, 0.99, 0.99, 0.96, 1.0, 0.98, 1.0, 0.99, 1.0, 0.98, 0.99, 0.98, 1.0, 1.0, 1.0, 0.99, 1.0, 1.0, 0.99, 1.0, 0.99, 0.99, 0.99, 0.98, 1.0, 1.0, 0.99, 0.99, 0.99, 1.0, 0.98, 0.98, 0.99, 0.98, 1.0, 0.99, 1.0, 0.98, 0.99, 0.99, 1.0, 1.0, 0.99, 1.0, 0.99, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.99, 0.99, 0.98, 0.98, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 0.99, 0.99, 1.0, 1.0, 1.0, 0.98, 0.99, 0.99, 0.99, 0.98, 1.0, 0.99, 0.99, 0.99, 1.0, 0.97, 0.99, 0.99, 0.97, 1.0, 0.98, 1.0, 0.99, 1.0, 0.99, 0.99, 1.0, 0.98, 1.0, 0.99, 0.99, 0.99, 1.0, 0.99, 1.0, 0.99, 0.99, 0.98]
test_acc: 99.090452%
In [10]:
train(train_time, 20, 30, 2e-06)
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1238: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1204: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1340: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
Epoch 21/30
 - 1451s - loss: 2.2175 - val_loss: 1.2477
Epoch 22/30
 - 618s - loss: 0.9582 - val_loss: 0.7107
Epoch 23/30
 - 618s - loss: 0.6167 - val_loss: 0.5100
Epoch 24/30
 - 618s - loss: 0.4461 - val_loss: 0.4011
Epoch 25/30
 - 620s - loss: 0.3659 - val_loss: 0.3359
Epoch 26/30
 - 618s - loss: 0.2927 - val_loss: 0.2830
Epoch 27/30
 - 619s - loss: 0.2496 - val_loss: 0.2524
Epoch 28/30
 - 619s - loss: 0.2245 - val_loss: 0.2293
Epoch 29/30
 - 619s - loss: 0.1972 - val_loss: 0.2059
Epoch 30/30
 - 618s - loss: 0.1784 - val_loss: 0.2014
In [12]:
train(train_time, 30, 40, 2e-05)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
Epoch 31/40
 - 630s - loss: 0.1487 - val_loss: 0.1430
Epoch 32/40
 - 620s - loss: 0.0975 - val_loss: 0.1137
Epoch 33/40
 - 623s - loss: 0.0733 - val_loss: 0.0949
Epoch 34/40
 - 620s - loss: 0.0582 - val_loss: 0.1035
Epoch 35/40
 - 622s - loss: 0.0500 - val_loss: 0.0942
Epoch 36/40
 - 621s - loss: 0.0426 - val_loss: 0.0836
Epoch 37/40
 - 622s - loss: 0.0392 - val_loss: 0.0826
Epoch 38/40
 - 624s - loss: 0.0328 - val_loss: 0.0779
Epoch 39/40
 - 622s - loss: 0.0310 - val_loss: 0.0739
Epoch 40/40
 - 636s - loss: 0.0260 - val_loss: 0.0685
In [10]:
train(train_time, 40, 50, 2e-05)
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1238: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1204: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1340: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
Epoch 41/50
 - 1647s - loss: 0.0236 - val_loss: 0.0671
Epoch 42/50
 - 620s - loss: 0.0208 - val_loss: 0.0707
Epoch 43/50
 - 621s - loss: 0.0164 - val_loss: 0.0625
Epoch 44/50
 - 638s - loss: 0.0217 - val_loss: 0.0744
Epoch 45/50
 - 635s - loss: 0.0206 - val_loss: 0.0851
Epoch 46/50
 - 637s - loss: 0.0191 - val_loss: 0.0613
Epoch 47/50
 - 639s - loss: 0.0169 - val_loss: 0.0718
Epoch 48/50
 - 637s - loss: 0.0152 - val_loss: 0.0586
Epoch 49/50
 - 635s - loss: 0.0097 - val_loss: 0.0622
Epoch 50/50
 - 638s - loss: 0.0160 - val_loss: 0.0608
In [10]:
train(train_time, 50, 60, 2e-05)
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1238: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1204: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1340: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
Epoch 51/60
 - 1659s - loss: 0.0087 - val_loss: 0.0606
Epoch 52/60
 - 621s - loss: 0.0077 - val_loss: 0.0743
Epoch 53/60
 - 615s - loss: 0.0150 - val_loss: 0.0834
Epoch 54/60
 - 618s - loss: 0.0133 - val_loss: 0.0690
Epoch 55/60
 - 614s - loss: 0.0120 - val_loss: 0.0654
Epoch 56/60
 - 619s - loss: 0.0111 - val_loss: 0.0604
Epoch 57/60
 - 621s - loss: 0.0109 - val_loss: 0.0626
Epoch 58/60

Epoch 00058: reducing learning rate to 1.9999999494757505e-06.
 - 615s - loss: 0.0097 - val_loss: 0.0737
Epoch 59/60
 - 615s - loss: 0.0073 - val_loss: 0.0521
Epoch 60/60
 - 616s - loss: 0.0060 - val_loss: 0.0471
In [10]:
train(train_time, 60, 70, 2e-06)
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1238: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1204: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1340: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
Epoch 61/70
 - 1568s - loss: 0.0026 - val_loss: 0.0494
Epoch 62/70
 - 616s - loss: 0.0025 - val_loss: 0.0533
Epoch 63/70
 - 616s - loss: 0.0045 - val_loss: 0.0493
Epoch 64/70
 - 617s - loss: 0.0050 - val_loss: 0.0500
Epoch 65/70
 - 617s - loss: 0.0047 - val_loss: 0.0500
Epoch 66/70
 - 617s - loss: 0.0046 - val_loss: 0.0506
Epoch 67/70
 - 616s - loss: 0.0041 - val_loss: 0.0529
Epoch 68/70

Epoch 00068: reducing learning rate to 1.9999999949504855e-07.
 - 616s - loss: 0.0041 - val_loss: 0.0493
Epoch 69/70
 - 616s - loss: 0.0042 - val_loss: 0.0473
Epoch 70/70
 - 616s - loss: 0.0044 - val_loss: 0.0453
In [12]:
train(train_time, 70, 80, 2e-07)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
Epoch 71/80
 - 628s - loss: 0.0014 - val_loss: 0.0474
Epoch 72/80
 - 620s - loss: 0.0041 - val_loss: 0.0504
Epoch 73/80
 - 623s - loss: 0.0038 - val_loss: 0.0493
Epoch 74/80
 - 620s - loss: 0.0037 - val_loss: 0.0498
Epoch 75/80
 - 621s - loss: 0.0039 - val_loss: 0.0489
Epoch 76/80
 - 621s - loss: 0.0039 - val_loss: 0.0491
Epoch 77/80

Epoch 00077: reducing learning rate to 2.000000023372195e-08.
 - 622s - loss: 0.0038 - val_loss: 0.0500
Epoch 78/80
 - 621s - loss: 0.0034 - val_loss: 0.0494
Epoch 79/80
 - 621s - loss: 0.0038 - val_loss: 0.0472
Epoch 80/80
 - 623s - loss: 0.0041 - val_loss: 0.0475

对测试集测试正确率

In [12]:
train(train_time, 100, 100, 5e-06, test_proc=True, use_epoch=12)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
[1.0, 0.99, 0.99, 0.98, 0.99, 0.98, 0.98, 1.0, 0.99, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 1.0, 0.97, 1.0, 0.98, 0.98, 0.99, 0.98, 0.99, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.97, 0.98, 0.98, 0.99, 1.0, 0.99, 0.99, 1.0, 0.97, 0.98, 1.0, 1.0, 0.98, 0.98, 0.99, 0.99, 1.0, 0.98, 0.99, 0.97, 0.98, 0.99, 1.0, 0.98, 0.99, 1.0, 0.99, 0.99, 1.0, 0.99, 0.99, 0.98, 0.99, 0.99, 1.0, 1.0, 0.98, 0.97, 0.99, 1.0, 0.99, 0.99, 0.99, 0.99, 0.99, 0.98, 0.98, 0.99, 0.99, 0.99, 0.97, 0.98, 0.99, 1.0, 0.98, 1.0, 0.98, 1.0, 0.98, 0.98, 1.0, 0.99, 0.99, 0.98, 0.98, 0.98, 0.99, 1.0, 1.0, 0.99, 0.99, 1.0, 0.99, 0.99, 0.96, 1.0, 0.99, 1.0, 0.98, 1.0, 0.97, 1.0, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.99, 1.0, 0.99, 0.99, 1.0, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 0.99, 0.98, 0.97, 0.99, 0.98, 1.0, 0.99, 1.0, 0.98, 1.0, 0.98, 1.0, 1.0, 0.98, 0.99, 1.0, 0.98, 0.99, 1.0, 1.0, 0.99, 1.0, 0.99, 0.98, 0.98, 0.99, 0.99, 1.0, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.98, 0.99, 1.0, 0.97, 0.98, 0.99, 0.99, 0.98, 1.0, 0.99, 0.99, 0.99, 1.0, 0.98, 0.99, 0.98, 0.97, 1.0, 0.98, 1.0, 0.98, 1.0, 0.99, 0.99, 1.0, 0.98, 1.0, 0.98, 0.99, 0.98, 1.0, 0.99, 1.0, 1.0, 0.98, 0.98]
test_acc: 99.000000%
In [11]:
train(train_time, 100, 100, 5e-06, test_proc=True, use_epoch=20)
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1238: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1204: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
WARNING:tensorflow:From D:\ProgramData\envs\tfdl_py35\lib\site-packages\keras\backend\tensorflow_backend.py:1340: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
[0.98, 0.99, 0.99, 1.0, 0.99, 0.99, 0.99, 1.0, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.96, 1.0, 0.97, 0.98, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 0.96, 0.98, 0.98, 0.99, 1.0, 0.99, 1.0, 1.0, 0.99, 0.99, 1.0, 1.0, 0.99, 0.98, 0.99, 0.99, 1.0, 0.99, 0.99, 0.97, 0.98, 0.99, 0.99, 0.98, 1.0, 1.0, 1.0, 0.99, 1.0, 0.99, 0.99, 0.98, 0.98, 0.99, 1.0, 1.0, 0.98, 0.97, 0.98, 0.99, 0.98, 0.99, 1.0, 0.99, 0.99, 0.98, 0.97, 1.0, 0.99, 0.99, 0.97, 0.99, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.97, 0.98, 1.0, 0.99, 0.99, 0.98, 0.98, 0.97, 0.98, 1.0, 1.0, 1.0, 0.98, 1.0, 0.98, 0.99, 0.96, 1.0, 1.0, 0.99, 0.99, 1.0, 0.98, 0.99, 0.99, 1.0, 0.99, 1.0, 1.0, 1.0, 0.98, 0.99, 1.0, 0.99, 0.99, 0.99, 0.98, 0.99, 1.0, 1.0, 0.99, 0.99, 0.99, 0.98, 0.96, 0.99, 0.99, 1.0, 0.99, 1.0, 0.98, 0.99, 0.99, 1.0, 1.0, 0.98, 0.99, 1.0, 0.99, 1.0, 1.0, 1.0, 0.98, 1.0, 0.99, 0.98, 0.98, 0.98, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.98, 0.97, 0.99, 0.99, 0.98, 1.0, 0.99, 0.99, 1.0, 1.0, 0.98, 0.98, 0.99, 0.97, 1.0, 0.99, 1.0, 0.99, 0.99, 0.99, 0.98, 1.0, 0.98, 1.0, 0.99, 1.0, 0.99, 1.0, 0.99, 0.99, 1.0, 0.99, 0.98]
test_acc: 99.050251%
In [13]:
train(train_time, 100, 100, 5e-06, test_proc=True, use_epoch=70)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
[0.97, 0.99, 1.0, 0.99, 1.0, 0.99, 1.0, 0.99, 1.0, 0.99, 0.98, 0.98, 1.0, 1.0, 0.99, 1.0, 0.97, 0.99, 1.0, 0.98, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 0.99, 0.99, 0.99, 0.97, 0.98, 0.98, 0.99, 1.0, 0.98, 0.99, 1.0, 0.97, 1.0, 1.0, 0.99, 0.98, 0.98, 0.99, 0.98, 1.0, 0.99, 1.0, 0.98, 0.99, 1.0, 1.0, 0.99, 0.99, 1.0, 0.99, 1.0, 1.0, 0.99, 0.99, 0.99, 0.98, 0.99, 0.98, 0.99, 0.97, 0.97, 0.99, 0.98, 1.0, 0.99, 0.99, 1.0, 0.99, 1.0, 1.0, 1.0, 1.0, 0.99, 0.97, 1.0, 0.99, 1.0, 0.99, 1.0, 0.97, 0.99, 0.99, 0.99, 1.0, 0.99, 0.98, 0.99, 0.98, 0.99, 1.0, 1.0, 1.0, 0.99, 0.97, 1.0, 0.99, 1.0, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.98, 1.0, 0.97, 0.99, 0.99, 1.0, 0.98, 1.0, 0.98, 0.99, 1.0, 1.0, 1.0, 0.99, 0.97, 1.0, 1.0, 1.0, 0.98, 0.99, 0.99, 0.97, 0.98, 1.0, 1.0, 0.99, 0.99, 1.0, 0.98, 1.0, 0.99, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 0.98, 1.0, 0.99, 1.0, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.98, 0.98, 0.99, 0.98, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.98, 0.99, 0.99, 1.0, 0.99, 0.98, 0.98, 0.99, 0.98, 0.98, 1.0, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 0.99, 1.0]
test_acc: 99.155779%
In [12]:
train(train_time, 100, 100, 5e-06, test_proc=True, use_epoch=80)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:60: UserWarning: Update your `Model` call to the Keras 2 API: `Model(outputs=Tensor("de..., inputs=Tensor("th...)`
[0.97, 0.99, 1.0, 0.99, 1.0, 0.99, 1.0, 0.99, 1.0, 0.99, 0.98, 0.98, 1.0, 1.0, 0.99, 1.0, 0.97, 0.99, 1.0, 0.98, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.99, 1.0, 0.99, 0.97, 0.98, 0.98, 0.99, 1.0, 0.98, 0.99, 1.0, 0.97, 0.99, 1.0, 0.99, 0.98, 0.98, 0.99, 0.99, 1.0, 0.99, 1.0, 0.98, 1.0, 1.0, 1.0, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.99, 0.99, 0.98, 0.98, 0.99, 0.98, 0.99, 0.97, 0.97, 0.99, 0.98, 1.0, 0.99, 0.99, 1.0, 0.99, 1.0, 1.0, 1.0, 1.0, 0.99, 0.97, 1.0, 0.99, 0.99, 0.99, 1.0, 0.97, 0.99, 0.99, 0.99, 1.0, 0.99, 0.98, 0.99, 0.98, 0.99, 1.0, 1.0, 1.0, 0.99, 0.97, 1.0, 0.98, 1.0, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.98, 1.0, 0.97, 0.99, 0.99, 1.0, 0.98, 1.0, 0.98, 0.99, 1.0, 1.0, 1.0, 0.99, 0.97, 1.0, 1.0, 1.0, 0.98, 0.99, 0.98, 0.97, 0.98, 1.0, 1.0, 0.99, 0.99, 1.0, 0.98, 1.0, 0.99, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 0.98, 1.0, 0.99, 1.0, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 0.99, 1.0, 1.0, 1.0, 0.99, 1.0, 0.98, 0.97, 0.99, 0.98, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.99, 1.0, 1.0, 0.98, 0.99, 0.99, 1.0, 0.99, 0.99, 0.98, 0.99, 0.98, 0.98, 1.0, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 0.99, 1.0]
test_acc: 99.155779%
In [23]:
train(train_time, 100, 100, 5e-06, test_proc=True, use_epoch=70, test_show=True)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:76: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
In [24]:
train_time = '20190330-003153'
In [15]:
train(train_time, 0, 20, 1e-04, simple_mode=True)
train nums: 60000
valid nums: 20000
test nums: 20000
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:79: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
the_input (InputLayer)          (None, 300, 64, 3)   0                                            
__________________________________________________________________________________________________
conv2d_25 (Conv2D)              (None, 300, 64, 16)  208         the_input[0][0]                  
__________________________________________________________________________________________________
batch_normalization_30 (BatchNo (None, 300, 64, 16)  64          conv2d_25[0][0]                  
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 300, 64, 16)  0           batch_normalization_30[0][0]     
__________________________________________________________________________________________________
max_pooling2d_16 (MaxPooling2D) (None, 150, 32, 16)  0           activation_30[0][0]              
__________________________________________________________________________________________________
conv2d_26 (Conv2D)              (None, 150, 32, 32)  2080        max_pooling2d_16[0][0]           
__________________________________________________________________________________________________
batch_normalization_31 (BatchNo (None, 150, 32, 32)  128         conv2d_26[0][0]                  
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 150, 32, 32)  0           batch_normalization_31[0][0]     
__________________________________________________________________________________________________
max_pooling2d_17 (MaxPooling2D) (None, 75, 16, 32)   0           activation_31[0][0]              
__________________________________________________________________________________________________
conv2d_27 (Conv2D)              (None, 75, 16, 64)   8256        max_pooling2d_17[0][0]           
__________________________________________________________________________________________________
batch_normalization_32 (BatchNo (None, 75, 16, 64)   256         conv2d_27[0][0]                  
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 75, 16, 64)   0           batch_normalization_32[0][0]     
__________________________________________________________________________________________________
max_pooling2d_18 (MaxPooling2D) (None, 37, 8, 64)    0           activation_32[0][0]              
__________________________________________________________________________________________________
reshape_6 (Reshape)             (None, 37, 512)      0           max_pooling2d_18[0][0]           
__________________________________________________________________________________________________
dense_11 (Dense)                (None, 37, 128)      65664       reshape_6[0][0]                  
__________________________________________________________________________________________________
batch_normalization_33 (BatchNo (None, 37, 128)      512         dense_11[0][0]                   
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 37, 128)      0           batch_normalization_33[0][0]     
__________________________________________________________________________________________________
gru1 (GRU)                      (None, 37, 256)      295680      activation_33[0][0]              
__________________________________________________________________________________________________
gru1_b (GRU)                    (None, 37, 256)      295680      activation_33[0][0]              
__________________________________________________________________________________________________
add_6 (Add)                     (None, 37, 256)      0           gru1[0][0]                       
                                                                 gru1_b[0][0]                     
__________________________________________________________________________________________________
gru2 (GRU)                      (None, 37, 256)      393984      add_6[0][0]                      
__________________________________________________________________________________________________
gru2_b (GRU)                    (None, 37, 256)      393984      add_6[0][0]                      
__________________________________________________________________________________________________
concatenate_6 (Concatenate)     (None, 37, 512)      0           gru2[0][0]                       
                                                                 gru2_b[0][0]                     
__________________________________________________________________________________________________
dropout_6 (Dropout)             (None, 37, 512)      0           concatenate_6[0][0]              
__________________________________________________________________________________________________
dense_12 (Dense)                (None, 37, 17)       8721        dropout_6[0][0]                  
__________________________________________________________________________________________________
the_labels (InputLayer)         (None, 11)           0                                            
__________________________________________________________________________________________________
input_length (InputLayer)       (None, 1)            0                                            
__________________________________________________________________________________________________
label_length (InputLayer)       (None, 1)            0                                            
__________________________________________________________________________________________________
ctc (Lambda)                    (None, 1)            0           dense_12[0][0]                   
                                                                 the_labels[0][0]                 
                                                                 input_length[0][0]               
                                                                 label_length[0][0]               
==================================================================================================
Total params: 1,465,217
Trainable params: 1,464,737
Non-trainable params: 480
__________________________________________________________________________________________________
Epoch 1/20
 - 374s - loss: 11.2944 - val_loss: 2.7388
Epoch 2/20
 - 366s - loss: 1.7732 - val_loss: 0.9688
Epoch 3/20
 - 366s - loss: 0.7053 - val_loss: 0.5218
Epoch 4/20
 - 366s - loss: 0.4031 - val_loss: 0.4148
Epoch 5/20
 - 366s - loss: 0.2679 - val_loss: 0.2929
Epoch 6/20
 - 366s - loss: 0.1877 - val_loss: 0.2365
Epoch 7/20
 - 365s - loss: 0.1363 - val_loss: 0.2300
Epoch 8/20
 - 366s - loss: 0.1012 - val_loss: 0.2190
Epoch 9/20
 - 366s - loss: 0.0801 - val_loss: 0.2194
Epoch 10/20
 - 366s - loss: 0.0613 - val_loss: 0.1961
Epoch 11/20
 - 366s - loss: 0.0529 - val_loss: 0.1843
Epoch 12/20
 - 367s - loss: 0.0466 - val_loss: 0.2015
Epoch 13/20
 - 370s - loss: 0.0371 - val_loss: 0.1826
Epoch 14/20
 - 366s - loss: 0.0317 - val_loss: 0.2491
Epoch 15/20
 - 366s - loss: 0.0297 - val_loss: 0.1982
Epoch 16/20
 - 365s - loss: 0.0280 - val_loss: 0.2468
Epoch 17/20
 - 378s - loss: 0.0239 - val_loss: 0.2394
Epoch 18/20
 - 398s - loss: 0.0224 - val_loss: 0.1907
Epoch 19/20
 - 390s - loss: 0.0208 - val_loss: 0.1791
Epoch 20/20
 - 389s - loss: 0.0184 - val_loss: 0.1675
In [25]:
train_time = '20190330-141738'
In [18]:
train(train_time, 0, 10, 1e-04, drop_out=0.6)
train nums: 60000
valid nums: 20000
test nums: 20000
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:79: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
the_input (InputLayer)          (None, 300, 64, 3)   0                                            
__________________________________________________________________________________________________
conv2d_31 (Conv2D)              (None, 300, 64, 32)  2432        the_input[0][0]                  
__________________________________________________________________________________________________
batch_normalization_38 (BatchNo (None, 300, 64, 32)  128         conv2d_31[0][0]                  
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 300, 64, 32)  0           batch_normalization_38[0][0]     
__________________________________________________________________________________________________
conv2d_32 (Conv2D)              (None, 300, 64, 32)  25632       activation_38[0][0]              
__________________________________________________________________________________________________
batch_normalization_39 (BatchNo (None, 300, 64, 32)  128         conv2d_32[0][0]                  
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 300, 64, 32)  0           batch_normalization_39[0][0]     
__________________________________________________________________________________________________
max_pooling2d_22 (MaxPooling2D) (None, 150, 32, 32)  0           activation_39[0][0]              
__________________________________________________________________________________________________
conv2d_33 (Conv2D)              (None, 150, 32, 64)  32832       max_pooling2d_22[0][0]           
__________________________________________________________________________________________________
batch_normalization_40 (BatchNo (None, 150, 32, 64)  256         conv2d_33[0][0]                  
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 150, 32, 64)  0           batch_normalization_40[0][0]     
__________________________________________________________________________________________________
conv2d_34 (Conv2D)              (None, 150, 32, 64)  65600       activation_40[0][0]              
__________________________________________________________________________________________________
batch_normalization_41 (BatchNo (None, 150, 32, 64)  256         conv2d_34[0][0]                  
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 150, 32, 64)  0           batch_normalization_41[0][0]     
__________________________________________________________________________________________________
max_pooling2d_23 (MaxPooling2D) (None, 75, 16, 64)   0           activation_41[0][0]              
__________________________________________________________________________________________________
conv2d_35 (Conv2D)              (None, 75, 16, 128)  73856       max_pooling2d_23[0][0]           
__________________________________________________________________________________________________
batch_normalization_42 (BatchNo (None, 75, 16, 128)  512         conv2d_35[0][0]                  
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 75, 16, 128)  0           batch_normalization_42[0][0]     
__________________________________________________________________________________________________
conv2d_36 (Conv2D)              (None, 75, 16, 128)  147584      activation_42[0][0]              
__________________________________________________________________________________________________
batch_normalization_43 (BatchNo (None, 75, 16, 128)  512         conv2d_36[0][0]                  
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 75, 16, 128)  0           batch_normalization_43[0][0]     
__________________________________________________________________________________________________
max_pooling2d_24 (MaxPooling2D) (None, 37, 8, 128)   0           activation_43[0][0]              
__________________________________________________________________________________________________
reshape_8 (Reshape)             (None, 37, 1024)     0           max_pooling2d_24[0][0]           
__________________________________________________________________________________________________
dense_15 (Dense)                (None, 37, 128)      131200      reshape_8[0][0]                  
__________________________________________________________________________________________________
batch_normalization_44 (BatchNo (None, 37, 128)      512         dense_15[0][0]                   
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 37, 128)      0           batch_normalization_44[0][0]     
__________________________________________________________________________________________________
gru1 (GRU)                      (None, 37, 256)      295680      activation_44[0][0]              
__________________________________________________________________________________________________
gru1_b (GRU)                    (None, 37, 256)      295680      activation_44[0][0]              
__________________________________________________________________________________________________
add_8 (Add)                     (None, 37, 256)      0           gru1[0][0]                       
                                                                 gru1_b[0][0]                     
__________________________________________________________________________________________________
gru2 (GRU)                      (None, 37, 256)      393984      add_8[0][0]                      
__________________________________________________________________________________________________
gru2_b (GRU)                    (None, 37, 256)      393984      add_8[0][0]                      
__________________________________________________________________________________________________
concatenate_8 (Concatenate)     (None, 37, 512)      0           gru2[0][0]                       
                                                                 gru2_b[0][0]                     
__________________________________________________________________________________________________
dropout_8 (Dropout)             (None, 37, 512)      0           concatenate_8[0][0]              
__________________________________________________________________________________________________
dense_16 (Dense)                (None, 37, 17)       8721        dropout_8[0][0]                  
__________________________________________________________________________________________________
the_labels (InputLayer)         (None, 11)           0                                            
__________________________________________________________________________________________________
input_length (InputLayer)       (None, 1)            0                                            
__________________________________________________________________________________________________
label_length (InputLayer)       (None, 1)            0                                            
__________________________________________________________________________________________________
ctc (Lambda)                    (None, 1)            0           dense_16[0][0]                   
                                                                 the_labels[0][0]                 
                                                                 input_length[0][0]               
                                                                 label_length[0][0]               
==================================================================================================
Total params: 1,869,489
Trainable params: 1,868,337
Non-trainable params: 1,152
__________________________________________________________________________________________________
Epoch 1/10
 - 644s - loss: 7.5910 - val_loss: 1.4838
Epoch 2/10
 - 620s - loss: 0.7873 - val_loss: 0.3734
Epoch 3/10
 - 619s - loss: 0.2956 - val_loss: 0.8165
Epoch 4/10
 - 624s - loss: 0.1561 - val_loss: 0.1244
Epoch 5/10
 - 612s - loss: 0.1002 - val_loss: 0.1405
Epoch 6/10
 - 612s - loss: 0.0765 - val_loss: 0.0855
Epoch 7/10
 - 612s - loss: 0.0541 - val_loss: 0.2091
Epoch 8/10
 - 610s - loss: 0.0459 - val_loss: 0.0897
Epoch 9/10
 - 611s - loss: 0.0403 - val_loss: 0.0664
Epoch 10/10
 - 610s - loss: 0.0313 - val_loss: 0.1092
In [19]:
train(train_time, 10, 20, 1e-04, drop_out=0.6)
D:\ProgramData\envs\tfdl_py35\lib\site-packages\ipykernel_launcher.py:79: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=Tensor("th..., outputs=Tensor("de...)`
Epoch 11/20
 - 627s - loss: 0.0374 - val_loss: 0.0617
Epoch 12/20
 - 613s - loss: 0.0266 - val_loss: 0.0987
Epoch 13/20
 - 612s - loss: 0.0303 - val_loss: 0.0662
Epoch 14/20
 - 613s - loss: 0.0234 - val_loss: 0.0631
Epoch 15/20
 - 613s - loss: 0.0220 - val_loss: 0.1973
Epoch 16/20
 - 613s - loss: 0.0208 - val_loss: 0.0833
Epoch 17/20
 - 613s - loss: 0.0168 - val_loss: 0.0447
Epoch 18/20
 - 613s - loss: 0.0165 - val_loss: 0.0795
Epoch 19/20
 - 612s - loss: 0.0144 - val_loss: 0.0839
Epoch 20/20
 - 611s - loss: 0.0177 - val_loss: 0.1269
In [ ]: